{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EKS CSI FSX Lustre Setup\n",
    "\n",
    "Amazon FSx for Lustre is a high-performance file system optimized for deep learning workloads. FSx provides POSIX-compliant file system access to S3 for multiple readers and writers simultaneously.\n",
    "  \n",
    "The Amazon FSx for Lustre Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the lifecycle of Amazon FSx for Lustre file systems.  \n",
    "\n",
    "* https://docs.aws.amazon.com/eks/latest/userguide/fsx-csi.html\n",
    "* https://github.com/kubernetes-sigs/aws-fsx-csi-driver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "iam = boto3.client('iam')\n",
    "sts = boto3.client('sts')\n",
    "cfn = boto3.client('cloudformation')\n",
    "eks = boto3.client('eks')\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "cluster_name = 'workshop'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Install the FSx CSI Driver for Kubernetes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create IAM Policy\n",
    "\n",
    "Create an IAM policy and service account that allows the driver to make calls to AWS APIs on your behalf."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \u001b[94m\"Version\"\u001b[39;49;00m:\u001b[33m\"2012-10-17\"\u001b[39;49;00m,\n",
      "    \u001b[94m\"Statement\"\u001b[39;49;00m:[\n",
      "        {\n",
      "            \u001b[94m\"Effect\"\u001b[39;49;00m:\u001b[33m\"Allow\"\u001b[39;49;00m,\n",
      "            \u001b[94m\"Action\"\u001b[39;49;00m:[\n",
      "                \u001b[33m\"iam:CreateServiceLinkedRole\"\u001b[39;49;00m,\n",
      "                \u001b[33m\"iam:AttachRolePolicy\"\u001b[39;49;00m,\n",
      "                \u001b[33m\"iam:PutRolePolicy\"\u001b[39;49;00m\n",
      "            ],\n",
      "            \u001b[94m\"Resource\"\u001b[39;49;00m:\u001b[33m\"arn:aws:iam::*:role/aws-service-role/s3.data-source.lustre.fsx.amazonaws.com/*\"\u001b[39;49;00m\n",
      "        },\n",
      "        {\n",
      "            \u001b[94m\"Action\"\u001b[39;49;00m:\u001b[33m\"iam:CreateServiceLinkedRole\"\u001b[39;49;00m,\n",
      "            \u001b[94m\"Effect\"\u001b[39;49;00m:\u001b[33m\"Allow\"\u001b[39;49;00m,\n",
      "            \u001b[94m\"Resource\"\u001b[39;49;00m:\u001b[33m\"*\"\u001b[39;49;00m,\n",
      "            \u001b[94m\"Condition\"\u001b[39;49;00m:{\n",
      "                \u001b[94m\"StringLike\"\u001b[39;49;00m:{\n",
      "                    \u001b[94m\"iam:AWSServiceName\"\u001b[39;49;00m:[\n",
      "                        \u001b[33m\"fsx.amazonaws.com\"\u001b[39;49;00m\n",
      "                    ]\n",
      "                }\n",
      "            }\n",
      "        },\n",
      "        {\n",
      "            \u001b[94m\"Effect\"\u001b[39;49;00m:\u001b[33m\"Allow\"\u001b[39;49;00m,\n",
      "            \u001b[94m\"Action\"\u001b[39;49;00m:[\n",
      "                \u001b[33m\"s3:ListBucket\"\u001b[39;49;00m,\n",
      "                \u001b[33m\"fsx:CreateFileSystem\"\u001b[39;49;00m,\n",
      "                \u001b[33m\"fsx:DeleteFileSystem\"\u001b[39;49;00m,\n",
      "                \u001b[33m\"fsx:DescribeFileSystems\"\u001b[39;49;00m\n",
      "            ],\n",
      "            \u001b[94m\"Resource\"\u001b[39;49;00m:[\n",
      "                \u001b[33m\"*\"\u001b[39;49;00m\n",
      "            ]\n",
      "        }\n",
      "    ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!pygmentize fsx/fsx-csi-driver.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !aws iam create-policy \\\n",
    "#     --policy-name Amazon_FSx_Lustre_CSI_Driver \\\n",
    "#     --policy-document file://fsx/fsx-csi-driver.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] Policy already exists.\n"
     ]
    }
   ],
   "source": [
    "with open('fsx/fsx-csi-driver.json') as json_file:\n",
    "    data = json.load(json_file)\n",
    "    policy = json.dumps(data)\n",
    "\n",
    "try:\n",
    "    response = iam.create_policy(\n",
    "        PolicyName='Amazon_FSx_Lustre_CSI_Driver',\n",
    "        PolicyDocument=policy\n",
    "    )\n",
    "    print(\"[OK] Policy created.\")\n",
    "\n",
    "except ClientError as e:\n",
    "    if e.response['Error']['Code'] == 'EntityAlreadyExists':\n",
    "        print(\"[OK] Policy already exists.\")\n",
    "    else:\n",
    "        print(\"Error: %s\" % e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:iam::231218423789:policy/Amazon_FSx_Lustre_CSI_Driver\n"
     ]
    }
   ],
   "source": [
    "account_id = sts.get_caller_identity()['Account']\n",
    "csi_policy_arn = 'arn:aws:iam::{}:policy/Amazon_FSx_Lustre_CSI_Driver'.format(account_id)\n",
    "print(csi_policy_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Kubernetes IAM Service Account\n",
    "\n",
    "Create a Kubernetes service account for the driver and attach the policy to the service account. Replacing the ARN of the policy with the ARN returned in the previous step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _The next cell runs for about 10min. Please be patient._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m[ℹ]  eksctl version 0.32.0\n",
      "\u001b[0m\u001b[36m[ℹ]  using region us-west-2\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 37.102353ms\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 82.788892ms\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 222.797624ms\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 352.276792ms\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 489.476128ms\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 1.802281056s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 3.149765568s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 5.359649408s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 9.312208128s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 19.662672384s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 48.844418048s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 1m2.069673984s\n",
      "\u001b[0m\u001b[32m[!]  retryable error (RequestError: send request failed\n",
      "caused by: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) from ec2metadata/GetToken - will retry after delay of 2m38.216507392s\n",
      "\u001b[0mrequest expired, resigning\n",
      "\u001b[36m[ℹ]  1 existing iamserviceaccount(s) (kube-system/aws-node) will be excluded\n",
      "\u001b[0m\u001b[36m[ℹ]  1 iamserviceaccount (kube-system/fsx-csi-controller-sa) was included (based on the include/exclude rules)\n",
      "\u001b[0m\u001b[36m[ℹ]  1 iamserviceaccount (kube-system/aws-node) was excluded (based on the include/exclude rules)\n",
      "\u001b[0m\u001b[32m[!]  serviceaccounts that exists in Kubernetes will be excluded, use --override-existing-serviceaccounts to override\n",
      "\u001b[0m\u001b[36m[ℹ]  1 task: { 2 sequential sub-tasks: { create IAM role for serviceaccount \"kube-system/fsx-csi-controller-sa\", create serviceaccount \"kube-system/fsx-csi-controller-sa\" } }\n",
      "\u001b[0m\u001b[36m[ℹ]  building iamserviceaccount stack \"eksctl-workshop-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa\"\n",
      "\u001b[0m\u001b[36m[ℹ]  deploying stack \"eksctl-workshop-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa\"\n",
      "\u001b[0m\u001b[36m[ℹ]  created serviceaccount \"kube-system/fsx-csi-controller-sa\"\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!eksctl create iamserviceaccount \\\n",
    "     --region $region \\\n",
    "     --name fsx-csi-controller-sa \\\n",
    "     --namespace kube-system \\\n",
    "     --cluster $cluster_name \\\n",
    "     --attach-policy-arn $csi_policy_arn \\\n",
    "     --approve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eksctl-workshop-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa\n"
     ]
    }
   ],
   "source": [
    "cf_stack_name = 'eksctl-{}-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa'.format(cluster_name)\n",
    "print(cf_stack_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'StackResourceSummaries': [{'LogicalResourceId': 'Role1', 'PhysicalResourceId': 'eksctl-workshop-addon-iamserviceaccount-kube-Role1-YQ8E1QVLFGWV', 'ResourceType': 'AWS::IAM::Role', 'LastUpdatedTimestamp': datetime.datetime(2020, 11, 21, 18, 10, 47, 955000, tzinfo=tzlocal()), 'ResourceStatus': 'CREATE_COMPLETE', 'DriftInformation': {'StackResourceDriftStatus': 'NOT_CHECKED'}}], 'ResponseMetadata': {'RequestId': '77ea8ffb-86c5-4354-92e7-b6ca0d0b8db7', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '77ea8ffb-86c5-4354-92e7-b6ca0d0b8db7', 'content-type': 'text/xml', 'content-length': '858', 'date': 'Sat, 21 Nov 2020 18:12:39 GMT'}, 'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "response = cfn.list_stack_resources(\n",
    "    StackName=cf_stack_name\n",
    ")\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eksctl-workshop-addon-iamserviceaccount-kube-Role1-YQ8E1QVLFGWV\n"
     ]
    }
   ],
   "source": [
    "iam_role_name = response['StackResourceSummaries'][0]['PhysicalResourceId']\n",
    "print(iam_role_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:iam::231218423789:role/eksctl-workshop-addon-iamserviceaccount-kube-Role1-YQ8E1QVLFGWV\n"
     ]
    }
   ],
   "source": [
    "iam_role_arn = iam.get_role(RoleName=iam_role_name)['Role']['Arn']\n",
    "print(iam_role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy CSI Driver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\n",
      "serviceaccount/fsx-csi-controller-sa configured\n",
      "clusterrole.rbac.authorization.k8s.io/fsx-csi-external-provisioner-role created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/fsx-csi-external-provisioner-binding created\n",
      "deployment.apps/fsx-csi-controller created\n",
      "daemonset.apps/fsx-csi-node created\n",
      "csidriver.storage.k8s.io/fsx.csi.aws.com created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -k \"github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=master\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Patch the driver deployment to add the service account that you just created, replacing the ARN with the correct role ARN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/fsx-csi-controller-sa annotated\n"
     ]
    }
   ],
   "source": [
    "!kubectl annotate serviceaccount -n kube-system fsx-csi-controller-sa \\\n",
    " eks.amazonaws.com/role-arn=$iam_role_arn --overwrite=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check S3 Bucket For FSX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = 's3://fsx-antje'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!aws s3 mb $bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           PRE code/\n",
      "                           PRE input/\n",
      "                           PRE model/\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-21 16:02:48          0 code/test_data/\n",
      "2020-11-21 16:02:48   18997559 code/test_data/amazon_reviews_us_Digital_Software_v1_00.tsv.gz\n",
      "2020-11-21 16:21:10      21708 code/train.py\n",
      "2020-10-30 18:14:13        615 input/data/test/part-algo-1-amazon_reviews_us_Digital_Software_v1_00.tfrecord\n",
      "2020-10-30 18:14:13        632 input/data/test/part-algo-2-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord\n",
      "2020-10-30 18:14:13      10728 input/data/train/part-algo-1-amazon_reviews_us_Digital_Software_v1_00.tfrecord\n",
      "2020-10-30 18:14:13      11812 input/data/train/part-algo-2-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord\n",
      "2020-10-30 18:14:13        679 input/data/validation/part-algo-1-amazon_reviews_us_Digital_Software_v1_00.tfrecord\n",
      "2020-10-30 18:14:13        642 input/data/validation/part-algo-2-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord\n",
      "2020-10-30 18:14:43          0 model/\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $bucket --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download Storage Class Manifest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -o storageclass.yaml https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning_s3/specs/storageclass.yaml\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get VPC ID and Subnet ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "export VPC_ID=vpc-02e8cba0a081bf4ad\n",
      "export SUBNET_ID=subnet-0a25103d821733ac0\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "source ~/.bash_profile\n",
    "\n",
    "#### Get VPC ID\n",
    "export VPC_ID=$(aws ec2 describe-vpcs --filters \"Name=tag:Name,Values=eksctl-${AWS_CLUSTER_NAME}-cluster/VPC\" --query \"Vpcs[0].VpcId\" --output text)\n",
    "echo \"export VPC_ID=${VPC_ID}\" | tee -a ~/.bash_profile\n",
    "\n",
    "#### Get Subnet ID\n",
    "export SUBNET_ID=$(aws ec2 describe-subnets --filters \"Name=vpc-id,Values=${VPC_ID}\" --query \"Subnets[0].SubnetId\" --output text)\n",
    "echo \"export SUBNET_ID=${SUBNET_ID}\" | tee -a ~/.bash_profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Security Group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "export SEC_GROUP_ID=sg-0ee7f41b8573a5d2b\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "source ~/.bash_profile\n",
    "\n",
    "export SEC_GROUP_ID=$(aws ec2 create-security-group --group-name eks-fsx-security-group --vpc-id ${VPC_ID} --description \"FSx for Lustre Security Group\" --query \"GroupId\" --output text)\n",
    "echo \"export SEC_GROUP_ID=${SEC_GROUP_ID}\" | tee -a ~/.bash_profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add an ingress rule that opens up port 988 from the 192.168.0.0/16 CIDR range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "source ~/.bash_profile\n",
    "\n",
    "aws ec2 authorize-security-group-ingress --group-id ${SEC_GROUP_ID} --protocol tcp --port 988 --cidr 192.168.0.0/16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update the environment variables in the `storageclass.yaml` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[94mkind\u001b[39;49;00m: StorageClass\n",
      "\u001b[94mapiVersion\u001b[39;49;00m: storage.k8s.io/v1\n",
      "\u001b[94mmetadata\u001b[39;49;00m:\n",
      "  \u001b[94mname\u001b[39;49;00m: fsx-sc\n",
      "\u001b[94mprovisioner\u001b[39;49;00m: fsx.csi.aws.com\n",
      "\u001b[94mparameters\u001b[39;49;00m:\n",
      "  \u001b[94msubnetId\u001b[39;49;00m: subnet-0a25103d821733ac0\n",
      "  \u001b[94msecurityGroupIds\u001b[39;49;00m: sg-0ee7f41b8573a5d2b\n",
      "  \u001b[94ms3ImportPath\u001b[39;49;00m: s3://fsx-antje\n",
      "  \u001b[94ms3ExportPath\u001b[39;49;00m: s3://fsx-antje\n",
      "  \u001b[94mautoImportPolicy\u001b[39;49;00m: NEW_CHANGED\n",
      "  \u001b[94mdeploymentType\u001b[39;49;00m: SCRATCH_2\n",
      "\u001b[94mmountOptions\u001b[39;49;00m:\n",
      "  - flock\n"
     ]
    }
   ],
   "source": [
    "!pygmentize fsx/storageclass.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create FSX Storage Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!kubectl delete -f fsx/storageclass.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "storageclass.storage.k8s.io/fsx-sc created\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f fsx/storageclass.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE\n",
      "fsx-sc          fsx.csi.aws.com         Delete          Immediate              false                  3s\n",
      "gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  36m\n"
     ]
    }
   ],
   "source": [
    "!kubectl get sc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Claim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!curl -o claim.yaml https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning_s3/specs/claim.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[94mapiVersion\u001b[39;49;00m: v1\n",
      "\u001b[94mkind\u001b[39;49;00m: PersistentVolumeClaim\n",
      "\u001b[94mmetadata\u001b[39;49;00m:\n",
      "  \u001b[94mname\u001b[39;49;00m: fsx-claim\n",
      "\u001b[94mspec\u001b[39;49;00m:\n",
      "  \u001b[94maccessModes\u001b[39;49;00m:\n",
      "    - ReadWriteMany\n",
      "  \u001b[94mstorageClassName\u001b[39;49;00m: fsx-sc\n",
      "  \u001b[94mresources\u001b[39;49;00m:\n",
      "    \u001b[94mrequests\u001b[39;49;00m:\n",
      "      \u001b[94mstorage\u001b[39;49;00m: 1200Gi\n"
     ]
    }
   ],
   "source": [
    "!pygmentize fsx/claim.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!kubectl delete -f fsx/claim.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "persistentvolumeclaim/fsx-claim created\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f fsx/claim.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\n",
      "fsx-claim   Bound    pvc-a7b6809e-e622-45d6-a754-d8b9d0ee6e25   1200Gi     RWX            fsx-sc         21m\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pvc fsx-claim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:          fsx-claim\n",
      "Namespace:     default\n",
      "StorageClass:  fsx-sc\n",
      "Status:        Bound\n",
      "Volume:        pvc-a7b6809e-e622-45d6-a754-d8b9d0ee6e25\n",
      "Labels:        <none>\n",
      "Annotations:   pv.kubernetes.io/bind-completed: yes\n",
      "               pv.kubernetes.io/bound-by-controller: yes\n",
      "               volume.beta.kubernetes.io/storage-provisioner: fsx.csi.aws.com\n",
      "Finalizers:    [kubernetes.io/pvc-protection]\n",
      "Capacity:      1200Gi\n",
      "Access Modes:  RWX\n",
      "VolumeMode:    Filesystem\n",
      "Mounted By:    <none>\n",
      "Events:\n",
      "  Type     Reason                Age                 From                                                                                      Message\n",
      "  ----     ------                ----                ----                                                                                      -------\n",
      "  Normal   Provisioning          11m (x3 over 21m)   fsx.csi.aws.com_fsx-csi-controller-55bcb55d5d-stdbn_34284e3c-2c25-11eb-8987-228ffe6b94e2  External provisioner is provisioning volume for claim \"default/fsx-claim\"\n",
      "  Warning  ProvisioningFailed    11m (x2 over 16m)   fsx.csi.aws.com_fsx-csi-controller-55bcb55d5d-stdbn_34284e3c-2c25-11eb-8987-228ffe6b94e2  failed to provision volume with StorageClass \"fsx-sc\": rpc error: code = DeadlineExceeded desc = context deadline exceeded\n",
      "  Normal   ExternalProvisioning  11m (x42 over 21m)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner \"fsx.csi.aws.com\" or manually created by system administrator\n"
     ]
    }
   ],
   "source": [
    "!kubectl describe pvc fsx-claim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Wait for status == Bound_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Update FSX to `autoImportPolicy: NEW_CHANGED`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fsx = boto3.client('fsx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = fsx.describe_file_systems()\n",
    "fsx_id = response['FileSystems'][0]['FileSystemId']\n",
    "print(fsx_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = fsx.update_file_system(\n",
    "    FileSystemId=fsx_id,\n",
    "    LustreConfiguration={\n",
    "        'AutoImportPolicy': 'NEW_CHANGED'\n",
    "    }\n",
    ")\n",
    "print(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
